SR WAR 


Systems  Center 
PACIFIC 


Technical  Document  3276 
September  2013 


A  Prototype  System  for  using 
Multiple  Radios  in  Directional 
MANET  (Mobile  Ad  Hoc  Network) 

A  NISE  funded 
Applied  Research  Project 


Chris  Cirullo 


Approved  for  public  release. 


SSC  Pacific 
San  Diego,  CA  92152-5001 


SSC  Pacific 

San  Diego,  California  92152-5001 


J.  J.  Beel,  CAPT,  USN  C.  A.  Keeney 

Commanding  Officer  Executive  Director 


ADMINISTRATIVE  INFORMATION 

This  report  was  prepared  for  the  Office  of  Naval  Research  (ONR)  by  the  Communications  and 
Networks  Department  (Code  55),  SPAWAR  Systems  Center  Pacific,  San  Diego,  CA.  The  project  is 
funded  by  the  Naval  Innovative  Science  and  Engineering  (NISE)  Program  as  an  Applied  Research 
project. 


Released  by 

T.  L.  Staley,  Head 

Tactical  Wireless  Networks  Branch 


Under  authority  of 
A.  Legaspi,  Head 
Networks  Division 


This  is  a  work  of  the  United  States  Government  and  therefore  is  not  copyrighted.  This  work  may 
be  copied  and  disseminated  without  restriction. 


SB 


EXECUTIVE  SUMMARY 


The  performance  benefits  of  using  high-gain  steerable  directional  antennas  for  wireless  networks 
are  well  known.  They  include  spatial  reuse,  higher  data  rates,  longer  range,  low  probability  of 
interception  and  detection  (LPI/LPD),  and  anti-jam  capabilities.  This  performance  is  desired  for 
backhaul,  line  of  sight  (LOS),  beyond  line  of  sight  (BLOS),  and  other  tactical-edge  wireless 
networks.  From  a  practical  stand  point,  it  is  difficult  to  employ  directional  antennas  in  a  mobile  ad 
hoc  network  (MANET)  as  most  current  radio  and  wireless  networking  protocols  were  designed  for 
use  with  omnidirectional  antennas.  The  fast  beam  switching  of  electronically  steerable  directional 
antennas  represents  a  physical  layer  change,  and  therefore  introduces  instabilities  to  many  protocol 
layers  of  a  networking  system  that  are  unaware  of  and  unable  to  quickly  adapt  to  physical  layer 
changes.  The  Multiple  Radio  per  Node  Network  Architecture  (MRNNA)  makes  it  is  possible  to  use 
electronically  steerable  directional  antennas  in  a  MANET  in  a  radio  agnostic  way  by  building 
networks  out  of  dedicated  point-to-point  links.  This  document  presents  a  protocol  that  accomplishes 
these  goals. 
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1.  INTRODUCTION  AND  BACKGROUND 

The  power  of  the  Multiple  Radio  per  Node  Network  Architecture  (MRNNA)  comes  from 
its  use  of  multiple  radios,  each  with  its  own  directional  antenna.  The  benefits  that  this 
provides  come  in  two  categories: 

1 .  Performance  benefits  that  are  derived  from  the  ability  to  create  multiple  links  per 
node  and  the  ability  to  minimize  interference  through  both  spatial  reuse  and 
frequency  separation:  Since  each  link  may  be  used  concurrently  at  its  full  link 
capacity,  total  network  throughput  will  be  greatly  increased  compared  to  single  radio 
systems  and/or  systems  that  use  omnidirectional  antennas. 

2.  The  benefit  of  being  radio  agnostic:  Previous  work  in  directional  wireless  networks 
has  focused  on  developing  new  protocols  that  are  aware  of  the  physical  layer  changes 
caused  by  beam  steering.  The  difficulty  in  this  approach  is  the  tremendous  amount  of 
complexity  in  synchronizing  the  functions  of  the  medium  access  control  (MAC) 
protocol,  routing,  quality  of  Service  and  beam  steering  all  operate  on  different  time 
scales.  The  protocols  that  have  come  out  of  this  research  are  inefficient  and  non¬ 
existent  in  commercial  radios.  Since  MRNNA  networks  are  built  out  of  dedicated 
point-to-point  links,  only  small  changes  to  the  physical  layer  needs  only  small 
changes,  which  occur  on  larger  time  scales.  Therefore,  standard  radio  protocols  can 
work  as  is  with  no  modifications.  A  caveat  of  this  is  that  convergence  times  of 
wireless  routing  protocols  have  an  impact  on  network  performance  just  as  in  other 
wireless  networks.  Since  the  MRNNA  protocol  has  perfect  knowledge  of  the 
topology  it  would  be  easy  to  synchronize  this  with  the  routing  layer. 

As  described  earlier,  prior  research  has  focused  on  modifying  network  and  radio  protocols 
to  support  directional  antennas.  The  paper  in  [1]  recognizes  the  shortcomings  of  directional 
MACs  and  proposes  a  more  practical  solution  that  involves  attaching  a  single  radio  to 
multiple-sector  antennas  and  then  selectively  switching  on  or  off  the  antennas  to  form 
topologies.  Such  a  system  could  be  considered  “radio  agnostic”  as  it  allows  the  use  of 
unmodified  MAC  protocols.  However,  as  a  single  radio  system  it  does  not  fully  exploit  the 
power  of  building  multiple  dedicated  links  and  also  provides  limited  support  for  mobility. 
References  [2],  [3],  [4],  [5],  [6],  [7],  [8],  [9],  [10],  and  [11]  describe  directional  MACs. 

The  Defense  Advanced  Research  Projects  Agency  (DARPA)  Wireless  Network  after  Next 
(WNaN)  radio  is  an  example  of  a  radio  with  multiple  transceivers  that  can  manage 
performance  frequency  to  form  topologies  and  get  performance  gains  from  the  reduced 
interference  due  to  frequency  separation.  The  WNaN  radio  was  not  designed  to  work  with 
directional  antennas  but  it  potentially  could  be  made  to  work  with  Directional  Ad-hoc 
Networking  Technology  (DANTEj/MRNNA  technology  with  some  effort. 

A  previous  paper  describes  the  MRNNA  and  the  technical  issues  with  using  directional 
antennas  [12],  In  this  report,  we  follow  on  that  work  and  present  a  link  selection  protocol  that 
we  have  implemented  in  a  prototype  system  using  unmodified  wifi  radios  and  electronically 
steerable  DANTE  antennas  developed  at  SPAWAR  Systems  Center  Pacific  (SSC  Pacific). 
We  will  discuss  the  functions  of  each  component  as  well  as  the  message  types  that  get  passed 
between  components  and  how  they  coordinate  to  form  a  topology.  We  will  also  discuss  radio 
issues  and  describe  the  basic  control  that  is  required  to  use  a  different  radio. 
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2.  MRNNA  SYSTEM  OVERVIEW 


A  MRNNA  network  node  has  a  central  master  controller  (MC)  that  connects  to  multiple 
sectors.  Each  sector  contains  a  single-board  computer,  a  radio,  and  a  directional  antenna. 
Figure  1  shows  a  diagram  of  a  MRNNA  network. 


Figure  1.  MRNNA  network  diagram. 

The  above  diagram  shows  how  it  is  possible  to  form  many  different  physical  topologies  in 
this  architecture  by  pointing  beams  in  different  directions.  As  nodes  move  around,  different 
choices  need  to  be  made  about  which  links  are  selected  as  “dedicated”  links  and  which  are 
ignored.  The  question  of  what  is  the  best  physical  topology  arises,  and  the  answer  to  this 
depends  on  the  goals  of  the  network.  Are  you  trying  to  maximize  throughput  between  a 
specific  pair  of  privileged  nodes?  Do  you  want  to  provide  more  robust  topologies  that  are 
more  tolerant  to  link  failures?  The  MRNNA  provides  a  rich  set  of  topology  choices,  and 
many  different  link  selection  algorithms  may  be  developed  and  tailored  to  specific  networks. 

Once  a  topology  is  selected,  the  configuration  of  links  will  be  maintained  for  as  long  as 
possible  to  provide  the  stability  that  allows  higher  layer  radio  protocols  to  function. 
Furthermore,  the  links  may  be  moved  to  different  frequencies  that  reduce  interference  and 
allow  each  link  to  operate  at  its  full  capacity.  Frequency  selection  can  be  treated  as  an  edge 
coloring  problem  for  which  polynomial  time  approximation  algorithms  exist.  However,  to 
minimize  interference,  treating  frequency  selection  as  a  pure  edge  coloring  problem  is 
impractical  as  it  ignores  real-world  radio-frequency  (RF)  phenomena  such  as  antenna  side 
lobes,  multi-path  bounces,  and  excessive  transmit  power.  A  form  of  interference  detection 
must  also  be  considered  to  mitigate  these  effects.  Power  control  is  an  option  to  reduce 
interference  but  it  is  not  something  we  currently  consider,  as  its  usefulness  really  depends  on 
the  specific  RF  environment  where  the  network  operates.  Much  of  our  prior  work  has  been 
on  establishing  and  maintaining  long-distance  links  on  moving  platforms  over  water.  This  is 
an  already  challenging  RF  environment  where  it  is  common  to  see  20+  db  fades  that  occur  on 
short  time  scales  due  to  the  “sea  bounce.”  In  this  type  of  network  where  the  nodes  are 
physically  distant  from  each  other,  the  interference  due  to  antenna  side  lobes  and  excessive 
transmit  power  are  less  of  an  issue.  In  networks  where  the  nodes  are  physically  closer 
together,  transmit  power  control  may  provide  more  of  an  interference  reduction  benefit. 

For  this  prototype  system,  we  built  seven  MRNNA  nodes  and  mounted  them  on  movable 
carts.  Figure  2  demonstrates  a  six-sector  node  with  a  staggered  triangle  design.  Using  these 
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carts,  we  can  move  nodes  around  and  see  how  they  make  the  correct  topology  decisions.  The 
rest  of  the  report  describes  the  function  of  each  component  and  how  they  all  work  together. 


Figure  2.  Six-sector  mobile  node. 

3.  SECTOR  CONTROLLER  FUNCTION 

The  main  functions  of  the  sector  controller  is  to  ( 1 )  control  the  beam  steering  of  the 
antenna  to  point  in  the  right  direction,  (2)  send/receive  HELLO  messages  in  each  direction  of 
its  coverage  area,  (3)  obtain  link-state  information  from  the  radio  and  report  it  to  the  master, 
and  (4)  perform  state  transitions  and  link  establishment/maintenance  operations  as  instructed 
by  the  master.  The  operational  states  of  the  sector  controller  are  as  follows: 

1 .  Discovery.  During  discovery,  the  sector  controller  will  sweep  through  the  beams  of 
the  directional  antenna  and  transmit  HELLO  messages  at  each  beam  position  for 
remote  nodes  to  hear.  HELLO  messages  contain  a  local  node  identification  as  well  as 
geo-location  information.  When  a  sector  controller  receives  a  HELLO  message  from 
another  node,  it  will  use  this  node  ID  to  obtain  link-state  information  from  the  radio 
and  report  the  link  to  the  master  controller. 

2.  Link  Establishment.  When  the  master  decides  that  a  sector  should  form  a  link  with  a 
remote  node  it  sends  an  LEO  message  to  the  sector  instructing  it  to  establish  a  link 
with  a  specific  remote  node.  A  sector  controller  in  the  LEO  state  will  begin  broadcast¬ 
ing  messages  to  the  remote  node  offering  to  create  a  link.  If  a  remote  sector  receives 
this  message  and  it  has  also  been  instructed  by  its  master  to  link  up,  it  will  respond 
with  an  LE2  message  that  is  an  agreement  to  create  the  link. 

3 .  Handover.  Due  to  mobility,  it  is  often  required  to  hand  over  a  link  from  one  sector  to 
another.  The  procedure  for  this  is  rather  simple.  Once  the  MC  runs  its  topology 
control  algorithm  and  determines  that  a  handover  is  necessary,  it  will  inform  the  new 
local  sector  to  pick  up  the  link  with  the  remote  sector.  It  will  also  inform  the  currently 
tracking  sector  that  a  handover  is  pending  and  it  should  be  prepared  to  change  states. 
The  reason  it  needs  to  inform  the  currently  tracking  sector  of  the  handover  is  to  avoid 
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race  conditions  and  miscommunication  in  the  rare  case  that  both  nodes  decide  to  hand 
over  at  the  same  time.  This  is  what  we  call  a  “double  handover.” 

4.  Tracking.  This  is  the  state  after  a  sector  controller  has  established  a  dedicated  link 
with  a  remote  sector  and  focuses  on  steering  the  beam  to  maintain  the  health  of  this 
link.  It  may  also  coordinate  a  frequency  switch  with  the  remote  sector  at  the 
instruction  of  the  MC. 

To  use  a  different  radio  in  the  MRNNA  node,  the  sector  controller  must  be  modified  so 
that  it  can  automatically  talk  to  the  new  radio  using  its  provided  control  interface.  The  extent 
to  which  this  is  possible  varies  from  radio  to  radio.  The  minimum  set  of  controls  the  sector 
controller  needs  is  for  the  radio  to  be  capable  of  switching  frequency  and  receiving  link-state 
data  such  as  Received  Signal  Strength  Indicator  (RSSI)  and  Link  Quality. 


Suoonsfui 

IE 


MCI  SC1-1  SC2-1  MC?  MCI 


DISCOVERY 

' 

TRACKING 


Figure  3.  Message  sequence  for  establishing  a  link  between  sectors. 


4.  MASTER  CONTROLLER  FUNCTION 

All  topology  control  decisions  are  made  at  the  master  controller.  Each  MC  is  connected  to 
multiple  sector  controllers  from  which  it  receives  link-state  information  and  geo-location 
information  of  remote  nodes  in  the  area.  The  MC  also  exchanges/forwards  link  state  and 
positional  information  with  other  directly  connected  MCs.  The  MCs  use  this  information  to 
build  an  adjacency  matrix  of  all  possible  links  that  may  be  formed  in  the  network.  After 
exchanging  tables,  all  MCs  will  have  an  identical  picture  of  what  the  network  looks  like. 
Since  they  all  run  the  same  link  selection  algorithm,  each  MC  will  compute  the  same 
topology  for  the  network.  Each  MC  then  examines  what  the  overall  network  topology  should 
be  and  compares  that  to  its  current  local  state  to  see  if  it  needs  to  effect  any  local  changes.  To 
execute  local  changes,  the  MC  assigns  its  sectors  to  the  appropriate  new  state  and  the  sector 
executes  the  change.  For  example,  if  the  MC  detects  that  its  local  sector  number  two  should 
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be  linked  to  remote  node  three,  it  will  send  a  link  establishment  message  to  the  local  sector 
and  the  local  sector  will  attempt  to  establish  a  link  to  a  sector  on  the  remote  node.  A  sample 
timing  diagram  for  the  full  link  establishment  sequence  is  in  Figure  3. 


5.  LINK  SELECTION  PROTOCOL 

The  link  selection  protocol  has  gone  through  three  major  revisions  in  its  development.  The 
first  iteration  was  named  Greedy  Local  Link  Selection  (GLLS).  In  this  version,  the  master 
controllers  made  decisions  about  which  links  to  select  based  solely  on  local  information.  It 
identified  the  remote  node  and  sector  based  on  information  embedded  in  the  beacons  and 
associated  an  RSSI  reading  for  each  remote  sector.  On  each  loop,  it  sorted  the  links  based  on 
signal  strength  and  assigned  its  local  sectors  to  track  a  specific  node  based  strictly  on  which 
ones  had  the  highest  signal  strength.  This  algorithm  was  simple  in  the  sense  it  takes  a  greedy 
approach  and  uses  local  information  only.  This  worked  well  for  networks  of  up  to  three  nodes 
but  instabilities  surfaced  after  adding  a  fourth  node.  The  issue  was  that  no  coordination 
existed  among  the  nodes.  Something  simple  such  as  a  sector  sweeping  through  its  beams 
causes  changes  in  the  physical  layer.  The  other  nodes  would  notice  the  change  and  not 
understand  what  caused  it.  Since  no  coordination  existed,  the  nodes  were  constantly  reacting 
off  of  each  other  and  the  network  would  become  unstable.  The  other  issue  is  that  asymmetric 
links  would  often  appear. 

The  next  version  of  the  protocol  was  the  GLLSI.  The  new  feature  in  this  protocol  was  the 
concept  of  “pair  bonding”  where  specific  sectors  of  two  nodes  would  communicate  and  agree 
to  pair  up.  This  added  stability  to  the  network.  The  new  problem  with  this  approach  was  that 
it  still  did  not  consider  the  entire  network  when  making  its  link  selections.  It  used  local 
information  and  information  obtained  from  its  one-hop  neighbor.  Through  mobility,  many 
complex  scenarios  emerged  in  which  the  nodes  did  not  have  enough  global  information  to 
make  a  proper  decision. 

This  led  us  to  the  MST-6  algorithm  in  which  the  MCs  exchange  information  with  each 
other  so  that  they  all  have  a  common  picture  of  the  network  at  a  point  in  time.  With  a 
common  picture  of  the  network,  they  can  each  independently  compute  the  same  topology  of 
what  the  network  should  be.  As  previously  mentioned,  many  different  link  selection 
algorithms  may  be  used,  and  the  best  one  depends  on  the  network.  As  a  base  case,  the  nodes 
connect  a  minimum  spanning-tree  using  link  distance  as  a  metric.  Then  we  add  as  many  links 
as  possible  to  fill  out  the  network. 


6.  NETWORKING 

One  of  the  main  benefits  of  the  MRNNA  architecture  is  the  separation  of  physical 
topology  decisions  from  the  MAC  layer  and  routing  protocols.  However,  use  of  multiple 
radios  can  potentially  complicate  the  job  of  routing  protocols  as  there  may  be  a  much 
larger  number  of  paths  between  nodes.  Each  node  has  six  sectors  and  a  master  controller, 
each  of  which  is  a  networking  device.  The  main  choice  to  be  made  is  whether  to  run  an 
instance  of  a  routing  protocol  on  the  sectors  or  to  put  them  in  “bridge”  mode.  Running  an 
instance  of  a  routing  protocol  on  the  sectors  increases  the  number  of  routers  on  the 
network  by  a  factor  of  6,  which  dramatically  increases  the  volume  of  protocol  messages 
and  also  increases  convergence  time.  Therefore,  bridge  mode  is  the  obvious  choice. 
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When  the  routing  protocol  chooses  a  different  path  than  the  one  chosen  by  the  topology 
manager,  one  issue  that  surfaces  is  the  presence  of  multiple  links  between  nodes.  This  can 
occur  when  nodes  are  close  enough  together  to  reach  each  other  using  the  side  lobes  of  the 
antenna.  To  achieve  full  performance  of  the  MRNNA,  it  is  necessary  for  the  routing  layer  to 
choose  the  same  link  as  the  topology  control  since  that  link  will  be  actively  maintained  by  the 
topology  manager  and  will  be  at  a  frequency  that  is  isolated  from  the  rest  of  the  nodes.  To 
achieve  this,  it  may  be  necessary  to  implement  some  limited  cross-layer  communication 
between  the  topology  manager  and  the  routing  protocol.  Another  area  where  a  cross-layer 
would  be  beneficial  is  to  adjust  routes  quickly  in  response  to  a  sector- to-sector  handover. 

We  have  tested  both  the  Open  Link  State  Routing  (OLSR)  protocol  and  BATMAN-ADV 
(better  approach  to  ad-hoc  networking  -  advanced)  in  a  four-node  MRNNA  network.  Both 
worked  and  performed  similarly.  Typically,  there  was  a  delay  of  4  to  6  s  for  the  protocols  to 
adjust  to  topology  changes.  This  delay  is  present  upon  establishment  of  a  new  link  and  more 
noticeable  to  the  user  on  sector-to-sector  handover  where  connectivity  may  cut  out  until  it  re¬ 
converges. 


7.  PERFORMANCE  TESTING 

As  previously  mentioned,  MRNNA  performance  benefits  come  from  the  ability  to  use 
multiple  links  between  nodes  concurrently.  We  demonstrated  this  concept  in  a  static  lab  test 
performed  with  five  nodes  (Figure  4).  The  number  of  radios  we  had  on  hand  limited  the  node 
count  for  this  test.  Figure  5  shows  test  results  for  two  runs  using  off-the-shelf  wifi  radios.  The 
first  run  uses  a  single  frequency,  so  all  radios  had  to  compete  for  access  to  the  channel.  As 
expected,  the  throughput  per  node  drops  as  we  add  nodes  to  the  network.  The  second  run  uses 
a  different  frequency  for  each  link  and  therefore  can  use  them  concurrently.  The  throughput 
per  node  increases  as  you  add  nodes  to  the  network.  Throughput  per  node  will  not  continue  to 
increase  like  this  forever.  It  really  depends  on  node  density,  the  number  of  radios  per  node, 
and  the  number  of  available  frequencies.  It  will  be  interesting  to  run  some  experiments  using 
a  higher  node  count  to  see  how  far  this  scales.  This  test  certainly  shows  the  immediate  benefit 
that  is  available  to  smaller  MANETs  such  as  in  Navy  ship-to-ship  LOS  networks. 


Node  5 


Figure  4.  Five  nodes  with  multiple  radios  on  different  frequencies.  Note  that  it  would  be 
possible  to  add  additional  links  without  increasing  node  count,  which  would  show  a  more 
dramatic  performance  increase. 
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Throughput  per  Node 


♦  Multiple  f  req 
■  Single  f  req 


Figure  5.  Results  of  running  iperf  on  each  link.  Throughput  per  node  is  calculated 
as  aggregate  throughput  divided  by  the  number  of  nodes. 

8.  CONCLUSION  AND  FUTURE  WORK 

It  is  possible  to  field  a  mobile  ad  hoc  network  using  directional  antennas  modifying 
wireless  network  protocols  by  using  the  MRNNA  approach,  and  many  different  topologies 
formed.  The  question  then  arises  about  what  is  the  “best”  topology.  The  answer  to  this  really 
depends  on  the  overall  structure  of  the  network  and  knowledge  of  which  traffic  flows  are 
more  important  than  others.  Many  optimization  problems  may  surface  such  as  choosing  a 
topology  and  selecting  multiple  routing  paths  to  maximize  throughput  between  a  pair  of 
nodes.  Although  it  is  desirable  to  avoid  cross-layer  design,  there  are  instances  where  some 
cross-layer  communication  between  the  topology  control  function  and  the  routing  layer 
minimize  traffic  interruptions  after  executing  a  change  in  physical  topology. 
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